Predicting Relevance based on Assessor Disagreement
نویسندگان
چکیده
We present the Predicted Relevance Model (PRM): it allows moving from binary evaluation measures that reflect a single assessor’s judgments, towards graded measures that represent the relevance towards random users.
منابع مشابه
A Discriminative Approach to Predicting Assessor Accuracy
Modeling changes in individual relevance assessor performance over time offers new ways to improve the quality of relevance judgments, such as by dynamically routing judging tasks to assessors more likely to produce reliable judgments. Whereas prior assessor models have typically adopted a single generative approach, we formulate a discriminative, flexible feature-based model. This allows us to...
متن کاملBetter than Their Reputation? On the Reliability of Relevance Assessments with Students
During the last three years we conducted several information retrieval evaluation series with more than 180 LIS students who made relevance assessments on the outcomes of three specific retrieval services. In this study we do not focus on the retrieval performance of our system but on the relevance assessments and the inter-assessor reliability. To quantify the agreement we apply Fleiss’ Kappa ...
متن کاملPredictors of Pre-diabetes Hemoglobin A1C Test in Overweight Obese and Overweight Employees Based on Protection Motivation Model
Background objective: This research paper aimed at to study the factors in relevance to the performance of screening behavior in per diabetes based on the stimulus model of preserve from the overweight and obese personnel of Karaj education administration. Materials and methods: the research method of the study is descriptive-analytical one and the statistical community of survey, are including...
متن کاملRelatively Relevant: Assessor Shift in Document Judgements
The evaluation of information retrieval systems relies on relevance judgements – human assessments of whether a document is relevant to a specified search request. In the past, it was demonstrated that test collection assessors disagree with each other to some extent on the relevance of documents and can be inconsistent in themselves. This paper describes a series of investigations on assessor ...
متن کاملUsers and Assessors in the Context of INEX: Are Relevance Dimensions Relevant?
The main aspects of XML retrieval are identified by analysing and comparing the following two behaviours: the behaviour of the assessor when judging the relevance of returned document components; and the behaviour of users when interacting with components of XML documents. We argue that the two INEX relevance dimensions, Exhaustivity and Specificity, are not orthogonal dimensions; indeed, an em...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2015